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Hello.
I'm Raven Alder and I'm here to talk to you guys
about denial of service and distributed
denial of service attacks and a
handy little Perl script that I've
written to track them through major ISP
backbones. Now,
I know most of you are probably pretty familiar with
this, but just a quick background on
denial of service and what we normally see
to let you know what the problem
is with ISPs and why a lot of them
don't slash can't address it.
A lot
of denial of service attacks that we're seeing
are...
Can you hear?
Okay.
Better?
Alright.
There are two major
types of denial of service attacks
that we see at my job. I work for
a major ISP. We'll see
denial of service attacks where they're
just coming from maybe one or two clients
with source IP addresses spoofed
or we'll see denial of service attacks
that are coming from like 200
clients with legitimate IP addresses.
There was actually a script
called DOS Tracker that someone at MCI
wrote a couple of years ago that
tracked the distributed denial of service
attacks, but it only tracked one IP address
at a time, so you'd have to run it repeatedly
in order to get anything
approaching tracking for modern DDoS attacks.
Now, the script that I've
written is called
Eyeball, and you can find it on
SourceForge, sourceforge.net
slash projects slash eyeball,
and it's capable of
tracking many IPs.
It's also capable
of tracking spoofed addresses.
Currently, it only works on
Cisco platforms. It does work okay on GSRs.
We hope to add support for
Juniper and Zebra routers
in the near future.
www.sourceforge.net
www.sourceforge.net
www.sourceforge.net
slash projects slash eyeball
slash projects slash eyeball
The script, as you can
happily see, is open source,
so you're totally free to take the source
code, modify it, change it. If you find
any bugs or problems, please let me know.
If you find any bugs or problems, please let me know.
If you find any bugs or problems, please let me know.
You can also send email
to me at
ravengeek
at users.sourceforge.net
Now, the problem that most ISPs
have in dealing with denial of service
attacks is that actually tracking down
anything with a spoofed address takes
absolute ages. You have to log into
each router in the path, so you usually
start at the router closest to the customer
being attacked. You can log in,
throw an access list on there on the
Cisco platform with the login
input keyword, which will log
where the actual source interface is
of the traffic coming into the router, and
then track it back. Most
links are point to point, so you
don't really have a lot of problems figuring out
what's on the other side of that link.
Now,
doing this
manually takes hours.
Doing this manually when you have
200 source addresses, most
ISPs won't even bother. And
this script obviously is not going to
change that. What it can do
is automate the process and give you
a list of, here are the IPs at which
this attack traffic entered your network.
Licensing for the script we covered
already. It's GPL. Have at it.
Also, a quick thank you to
Ben Stern, Tiffany Mork, Steve Ains,
and most of all, my husband Raven Black
for much help getting this up
and running and tested and all that.
In order to run this script,
it's a Perl script. You obviously need a Perl
environment wherever you're set up.
You will have to set up little access files
with your username and passwords. On
Cisco's, you will usually require
a regular password
to just log on via Telnet, your
VTY passwords, and
an enable password, or enable
secret. The file that you'll
set up can handle
TACACs or things where you have an actual username,
whether it's just TACACs for the
login and then regular
local database passwords for the enable, or
whether it's TACACs the whole way.
If you have your router set up so that you
can have more than one authentication method,
i.e. a TACAC server and then fall back to
local bases, you just run the little script
to set up your file twice
and then append the second onto there.
Bundled with
the package, which you can get on SourceForge,
is a script called
RouterAccessEncode.
You invoke that
with output, the little angle
bracket output, towards a file
called RouterAccess-
default.
It will prompt you for
your username, your login
password, and your enable password.
This uses the
Telnet Cisco module in Perl,
which will intelligently handle
logins, so if you give it a username
and the router happens to be configured to
only authenticate from a local database and doesn't
use usernames, then it won't return
the username for the password or anything like that.
It should work just fine. If you have a
backup method, run the script twice,
but the second time, append the output
to RouterAccess-
default, and that will
put in, if it fails on the first
authentication, try again with the second set of
username and passwords.
This should handle RADIUS, I haven't tried
it with that, but it works okay on
TACACS and local passwords.
Alright, now
once you've set up this script
there, the .routeraccess
default file has to be located
in your home directory to actually work.
All these files that
Eyeball uses should be located in the
same directory, but that doesn't matter
whether it's your home directory or not. The only thing that has to be
in your home directory is the .routeraccess-default.
Also,
the encryption that's done on
the .routeraccess-default is
really, really weak. It's almost
like the, you know,
ROT13 or something. So it's only there
to prevent a casual observer who happens
to look at your console
or something from knowing your passwords.
Do not store this with permission
so that anyone else can read it because
they can decrypt it with a
napkin and a pen.
Alright, so
after you have created the proper
.routeraccess-default
file, you will also
have to set an environment
variable for the
Eyeball access list. Now
the way that this script works is it will
go into each router in your network and create
an access list with the same number.
The way that Cisco does access lists, they have
their access list grouped by number
and the number represents the type of access list it is
so anything between 101 and
199 is going to be an extended IP
access list. That's what you want to use.
You have to make sure that this access list is
not used on any other router in your network.
Otherwise, when the script logs
in and tries to create the access list,
depending on the version of iOS and the bugs therein,
it may overwrite, it may change,
it may just totally delete your other access list
and screw up your routing otherwise.
So, bad idea. The script,
if you do not have this set, will
actually kick up an error message and say
you need to set an environment variable for
our eyeball access list.
So, just find an unused number between 101
and 199, you're all set.
Also, one of the
things that was originally
somewhat challenging is
the router needs to know what is on
your network. Assuming that you're
an ISP or a provider otherwise,
you don't want the script attempting to log into
the routers of your peers, etc.
So, just crawling through the network and
going interface by interface until your passwords
fail is not really a good idea.
It tends to annoy people.
So, you have to create a list of IP addresses
or of host names
that you own, essentially,
that the script is allowed to attempt to log into.
And we've also included here
a script
that will create that list for you.
Most ISPs that I know
have a list of their routers,
so that when they want to make some sort of policy change
they can just go through a script to change
to all the routers.
So, the eyeball make interface list
.pl script
will take care of that for you.
It takes the list of your routers as input
and you have to direct the output
to a file called
eyeball-valid-routers.txt
If you are
constantly adding things to your network
and removing things from your network,
you may want to cron the script so you run it
once a day, at night,
during your off-peak hours, whenever.
So that you always have a current list of IP addresses
that the script can log into
for your routers.
mile-of-router-list.txt
There's just whatever you call your
input list.
Alright, so once you've run these two scripts
you'll have everything to
set the environment
variable for eyeball access list.
Oh, one other thing that should
be set.
The eyeball wait time variable.
It's set to default to 10
seconds and it's just how long that access
list will remain on the router
before being pulled off.
So how long it will be there trying to log the attack packets.
10 seconds
is the default, but
if your routers run any sort of fast switching
Cisco express forwarding and things like that
we have seen problems with
Cisco's implementation because
fast switch packets do not get written into the
system log.
So if you're dealing with an attack
where it's like one packet a minute or something
then you'll want to expand the
eyeball wait time variable
to leave the list on there for maybe two minutes
or something to catch each packet.
Now, this is going to slow down the run of the script
because every router you log into
you have to wait that much time for it to catch
the packets as well as all the logging in
chewing through the output time, etc.
There are two
more debug variables that
you can set if you want to actually see
output.
Eyeball ultra debug and eyeball
insane debug. Eyeball ultra
debug, if you just set it to one
it will turn on debugging output so that you can
see what the commands you put
into each router are.
You can also see some helpful comments that I've thrown in the
script.
Eyeball insane debug will turn on what the router sends back
to you. That's a whole lot of output.
It's like a show log of each
router that you go through. You'll see the
show ints in the
show frame of the map, etc.
Lots and lots of stuff.
So that pretty much covers
the prerequisites. Oh, also
on your routers for this to work. Most people
do, but you will have to have
logging buffered debugging or logging
buffered informational turned on so that there is
a local syslog on the router for the script
to write to. If there isn't,
it does not go out to your syslog daemon
and parse through the logs or any such things.
Using the script itself,
it takes as
command line arguments the router that you want
to start tracking on, which will almost always be
the router right next to the customer
being attacked. The IP address
that is being attacked. Currently the script only
supports one IP address
being attacked at a time.
Could expand that in the future.
And you can also specify the type of
traffic. IP, TCP, ICMP,
UDP, what have you.
We will probably
be including, specifying the
port numbers in future, but the script
doesn't do that at the moment.
IP is the
default type of attack traffic if you do not
specify this. This
has caused problems in some implementations
if the customer is being ping flooded
and you just look for IP traffic
Thank you.
So if it is a ping flood attack
make sure you set it to ICMP.
While the script is running, you should
be seeing messages printed to your display
letting you know which routers that
the script is logging into, what it is
seeing in the output, whether it found
packets or not, and what the routers that it is going through.
The script is multi-threaded, so
if you are coming in through several
different places off of one router, it will
branch at that router and start
running the script.
Or rather, start logging into
the several routers from each individual point.
Blame for this is mine,
so comments, questions, suggestions,
bug reports, etc. send to
ravengeek at usersourceforge.net
Alright, let's get to
the script.
It uses a module called
router access which is included
with the distribution of the script
and all that router access does is
throws the login functions on there
you'll see them called
a bit later.
It will take the router, the
attacked address, and the packet type as arguments
and you can specify the wait time
as an environment variable. It does not check that
from the command line.
You'll get an error message there.
You don't have eyeball access list set
as an environment variable.
You can see we have
a function here to actually
throw on the
access lists on the router
and to remove it.
Simple Cisco commands that you guys are probably
really familiar with. You'll notice the host
keyword in the access list. This is because
it only goes through one
attacked IP address at a time.
If you
have several addresses being attacked,
then I would suggest running it
at least until we put support in for that
running it once for each IP being attacked.
The script will output
to eyeballoutput.txt in the directory
that you have invoked it from with all your
eyeball stuff in it. I would move
that after I have done each
individual address being attacked so it
doesn't overwrite append any such thing.
We also see the
array here to actually take commands off
the router.
Now when it first
tries to log into a router, it will
print an error message to the console if it
actually can't log into the router, otherwise
we'll give you, I've logged in, it took
this many seconds to this particular router.
It uses
the shover command to distinguish
between a regular Cisco router
and a Cisco gig switch router
because the logging format and a couple of other
things are different on the gig switch routers.
If you actually look at the show log
on the output for a gig switch router,
you will see at the beginning of the line
slot 7 and then later on
pos 0 for pos 7 0
on a line card.
So shover identifies
the difference between a gig switch router
and a regular one. I believe
that, I'm not real savvy with junipers
but I've been told that shover will also work on
a juniper so when we start including those in
we'll probably also differentiate
via junipers that way.
So,
now in order to determine
which way the
which interface the attack traffic is going out
it will do a show ip
route on the address
that's being attacked.
If this is not something directly connected
then it will just do a show ip route on the next
hop repeatedly until it gets to something that
is directly connected. This is because
when we put the access list on the interface
we're going to apply the access list
outbound on the interface that is
facing the router being attacked.
We can use the log
input command on there to log
where it's coming in from. This gives you
a relatively foolproof way of tracing spoofed
packets. The script does
treat every packet that it looks for
as a spoofed packet. So it doesn't attempt
to determine whether the packet header
is spoofed or not. It'll just
do a show ip route and log
input on that interface regardless.
Alright.
Here we see the access
list actually being applied.
IP access group, the access list number
to the interface. It will sleep for
whatever the wait time is set to which is
whatever you set your eyeball wait time
environment variable to. Defaults to 10 otherwise.
One thing that
you may want to
look at if you do not have
fast switching available
if you have a packet sniffer anywhere
near the guy being attacked, or you can
just throw an access list that permits IP
any any or permits ICMP any any
and logs. Try and diagnose
before you actually run eyeball the type
of traffic that's coming through and how often
you're seeing it. It won't really help
you very much to use the
default wait time of 10 seconds if you're
only getting a packet a minute. You've got
a 1 in 6 chance of your 10 seconds being
that 10 seconds.
One bug that we have encountered.
What you should be seeing in the
show log of your router after you've
run the script in each router that the script
passes through, you will see three
configured from
whatever interface you've come in on whether it's
console or VTY or what have you
by username if you have
or anything with the username enabled. Three
separate times. Now the way that
the script works is it will log into
the router, create the access list,
get out of global config mode, go back into
global config mode, apply the
access list to the interface with access
group whatever out.
Wait for the eyeball wait time,
remove the access list, get out of global
config mode, and then later gets back
into global config mode to remove
the access list from the router entirely
and then gets out again. So you should see three
configured from
console by Raven, etc.
And the packets that are logged
should be after the second
iteration of that.
There seems to be some
problem with the timing of this
by Cisco and I've talked
to their attack about this, but
there are a good number of times where you'll see
configured from console, configured from console, configured
from console, anything that happens for the
next minute or so, and then all the
logged packets.
So it appears that the
packets are being logged after the
access list has not only been removed from
the interface but has been removed from
the router entirely. It's kind of strange.
So we put an extra
little sleep function in there so
that if the access list is
removed you still wait a bit before
you actually check the logs to see if anything
happens. If you have this problem
occurring, then check
your version of IOS.
It was happening a good bit in
12.0.16s but seems to be fixed in
12.0.16st. You may also
want to talk to Cisco about it.
They are aware that the script exists.
Alright, here's the
log parsing bit. The
Sec6 IP access you should see
in anything that is reporting
the logged packets.
You'll see the access list number there
and the interface that it came in on.
You will also
see the interface info
on there.
If you have an ATM
interface, then you may see
VPI, VCI information. If you have a frame
relay interface, you'll see DILCs, things like that.
All that gets pulled by the script.
The dreaded
error message that you see
all too often if you have fast switching
and so all packets are not being logged.
No packets seen in router.
If this happens, the script will
just stop at that router and not attempt to search
anything further upstream. It wouldn't know
where to look. The only
fix that I have found so far
for that, if the fast switching is
enabled and so all packets aren't being logged,
is you can try and run the script again
or you can
increase the wait time. Sometimes that helps.
It will then
do a
look up in the correspondence between
the IP address and
the ATM or frame relay address is there.
It defaults to getting
all sorts of layer 2 information
regardless of what
is actually on the router. It's a lot easier
than actually trying to determine, okay this is
an ATM interface, we'll treat it like that. It just gets all the
information. So if you see lines in your output
of show ATM map
and there's an erroneous command because
you haven't got any ATM interfaces on that
particular router, that's okay.
It's in the script. That's supposed to happen.
And it will then log out of the router
and move on to
the next one.
Now I've done some testing of this
script and
it seems to work fairly well
but fast switching is the
biggest problem that we have.
It has to be run by someone with
enable access to the backbone
obviously because you need the enable
scripts to be put into the input file.
This means that end users
cannot run this script.
Pretty much only ISPs are capable of doing
that.
If any of you feel
like giving it a try, please
let me know how it works, what happens, etc.
I have tested this on Linux
platforms and on Solaris platforms.
Everything works okay on
both of those as long as you have the proper
cables installed. I haven't tried it under
the Windows implementation of Perl. I'm not much of a
Windows chick, but
it should in theory work.
Any questions?
Yeah?
You mean if your customer is multi-homed
from one particular router?
Yeah.
If the customer has multiple links
into your core, then you should run
iBall twice or
for as many times as each link they have starting at
each router that they have a link to you
on. So if they have
two T1s out of one router in
you can start it on that one router and it
will be fine. If they have two T1s
into different routers, run it once starting
at each of those different routers and it should pick
up everything.
Okay.
Well, the output
in the log will tell you that packets
are going to the customer's IP
address. You don't give it the customer's
MAC address or anything like that.
So if you have multiple routes
to the customer from your one particular
network, it doesn't matter
which one. It will just do show IP route
and determine the interface there.
BGP gets kind of strange
there because if you're using BGP, which
like most everyone on the internet is,
then BGP will only see one
quote-unquote best path and it will see
but that's the path that the traffic would be taking.
So I don't think it's
a huge problem there.
Any other questions?
Yes.
Well, having the
enabled passwords, obviously you shouldn't have
this sitting around on your home machine
sending things via
totally unencrypted telnet over the
internet. Cisco, unfortunately, doesn't support
secure shell yet, but it's very important
to protect the file that
has your
the router access default, etc.
You should set it so that its permissions, if you're
on a Unix system, are only readable to
you. It is a
concern of mine, yeah.
As far as the
entire network topology
is not actually saved in the eyeball
valid routers, it's just a list of IP
addresses. It doesn't tell you what connects to what.
It just says, these are the IP
addresses that I am allowed to try and connect to.
Yes?
No, I haven't
had that come up. The question was, have you
run into problems where you are unable to remove
the access list in the same telnet session?
It may be an iOS specific thing,
but it seems like
if it actually manages to
log into the router, it can
throw on the access list, look for traffic.
The biggest problem I've seen is when it doesn't
actually log any packets because of the fast switching
stuff, but I've never actually had it die
like mid-session.
If it does, though, one of the
things that would be fairly helpful is with the
output that it puts to the screen, you can see which
routers it's logging into, and
you will have one particular access
list that you would only use for this. That's why it
needs the eyeball access list variable.
So, you could, worst comes to
worst, go back through the routers that it's out
and put on the screen and make sure that those
IP access lists have been removed from the
routers.
No, I haven't seen that.
The output...
This is console on the
Cisco or console on your Unix box?
On the Cisco? Because the script is
run from a Unix box and just does the
telling it into each, so it doesn't display
the entire output on your
console. Hopefully, all it
would do is... If you have a
particular length of your
log buffer on the router, then it
can just... It would... Worst comes to worst,
it gets wrapped.
Does that... Is that what you're saying
freezes up your TTY?
No, no terminal monitoring at all.
You just do the show log, you parse whatever's in the
log, so if it's overwritten its log buffer
18 times, you're only going to get that last
screen of information. So in case
of like a truly massive DDoS attack
or something like that, then you probably
have to take that list that you
got of the attacking addresses,
try and deal with those, whether calling up your
ISPs and your peering ISPs
and telling them about them, shutting off those that are your
customers if you are the ISP, etc.,
and then running the script again to get the next
lump of them, because it's only going to pull what's in the
syslog, or rather what's in the
buffered log.
Yeah.
Yeah, if it gets to the point that it's
killing your router, then the script
will probably aid in your router's
death slightly, but
you're already killed.
We did
try running it on some routers that were
short on memory, and indeed in
one of the initial sketches out
for the layout of the script,
I was going to have it do like
a couple of show runs, pull out secondary
IP addresses for the interfaces so that if you
had an Ethernet interface with two IPs assigned
to it, it would include both of those in
the allowed list.
As it is now, it will only include whatever
is in a show IP interface brief.
So that's only the primary IP address
on any interface, but it will get IPs
of sub-interfaces and such.
But
we have a couple of routers in the network that I was
testing on that are very short on memory,
and so avoiding doing a show run
was generally a good idea.
So as to avoid causing malloc failures.
Yes?
It does do the ARP addresses.
That's why it pulls the
show frame relay map, etc., etc., to get the ARP,
and then when it goes through the logging,
it pulls both the
interface number and the
interface type, so it'll get DILSI,
VPI, VCI, what have you.
I haven't actually tried it on
any multipoint links.
It works fine pulling the information.
I get the information there, but the network
that I tested it on only has
point-to-points in its core.
So if anyone has multipoint links,
you know, some sort of funky ATM setup
or what have you, I would love to hear how it works.
Yes?
.
I don't have it with me,
but it's really simple looking.
It's just a list of IP addresses
and the interface on your router
where it came from.
So new line separated text file.
Yes?
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It hasn't been extensively tested yet, so.
Okay.
Yeah.
Okay.
So, if the router is one of our routers,
then it will open up a temp file.
If it can't open the temp file,
it dies with an error message,
and
it will go through,
it will fork and create children processes for each,
bless you,
individual router,
and
and it will just create a separate little session
for each one of those.
It will wait to actually close out the whole process
it counts how many children processes it has
at any given point.
And it will wait for that to close out the whole thing,
but essentially, I am not the queen of multithreading
this is essentially the only way that I know how to do it.
I know more about routers than I do about Pearl.
So, if you have a suggestion to improve efficiency
or technical performance,
then send it to me an email.
Anyone else?
Yes?
Yes?
Well, I've tested it on routers that have interfaces
up to OC48 speeds that were at about,
the OC48s were on 20% utilization,
but on the same routers we also had OC3s
that were at like 80 or 90% utilization,
and it didn't seem to cause a noticeable performance lag
on the routers or anything.
Thank you.
As far as attack traffic volumes,
I haven't seen anything above about,
let's see, I didn't test it with high level traffic volumes
with more than about 200 attacking addresses
in the last couple of versions of the script,
simply because the ISP that I work for
has not had an attack of that level going through it
in the later versions of the script.
It seemed to work all right in earlier versions,
but a good number of things have been changed.
I've gone up to about
10, 15 attacking hosts sending about
100 megs of traffic or so to the
attacked address, and the script handled that okay.
Other questions?
All right, we're down early.
.
Thank you.
